New Feature Sets for Summarization by Sentence Extraction

they don’t necessarily provide a coherent account— be used as a basis for further processing. Ideally, the document would be thoroughly analyzed using linguistic and world knowledge to determine which sentences are appropriate for the extract. In practice, the necessary analysis is still too immature or too computationally intensive to yield sufficient results. Many existing systems extract sentences on the basis of a limited set of mundane features. Still, the features that are most often used tend to be based on notions about document structure (for example, sentence position within the document, sentence length, repetition of words from the title or headings, selected cue words or phrases) or information content (for example, presence of high-frequency content words). The sidebar “Automatic Summarization” provides more details. I wanted to investigate whether I could enhance the feature-based extraction strategy by including some features that weren’t primarily information retrieval oriented. The features I wanted to try were originally developed to recognize writing style and so were somewhat geared toward reducing the influence of subject-specific (IR) elements. The underlying idea is that, when you try to summarize an article through sentence extraction, you’re assuming that the most important information is concentrated in specific sentences. If this is indeed true, the article’s author, knowing which sentences are the most important, might have consciously or subconsciously written these sentences in a different style (measurable, recurring patterns in the usage of vocabulary, grammar, text structure, and so on) from the rest of the article. If true, this supposition would likely allow valuable additions to the sentence extraction toolbox. People study automatic writing-style recognition primarily in the context of authorship attribution. In this task, you examine a certain text and try to determine which out of a given group of authors wrote it.4 The decision is based on information about several style markers, or features, such as vocabulary size or the distribution of a small set of specific vocabulary items. You generally learn about the markers from inspecting other texts by the same authors. One main focus of authorship attribution research is to create an inventory of useful style markers.5 Another important focus is to develop techniques for using these style markers to provide reliable-enough probability estimates for each potential author.6 I developed a new technique for estimating probabilities as well as several sets of style markers that complement this technique. I wanted to find out if this technique and these features could also be used to locate extractable sentences in a document.